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1.  Introduction 


Fusion3D  is  a  software  application  for  stereoscopic  visualization  of  3-D  terrain 
data.  It  was  developed  in  the  Image  Processing  Branch  at  the  US  Army  Research 
Laboratory  by  Dr  lohn  F  Dammann,  Jr.  It  uses  a  3-D  display,  3-D  glasses,  and  a 
3-D  mouse  to  quickly  view  province-sized  3-D  maps  in  stereo.  Its  robustness  and 
flexibility  allow  it  to  ingest  large  3-D  data  sets  from  a  variety  of  sources,  and  it 
incorporates  many  useful  features  to  aid  a  user  in  the  exploitation  of  3-D  terrain 
data.  Many  of  those  features  were  intended  to  provide  users  with  improved 
situational  awareness  and  aid  in  mission  planning;  however,  recently,  in  an  attempt 
to  further  improve  situational  awareness,  Fusion3D  was  modified  to  support 
overlaid  real-time  data  onto  the  3-D  terrain  data.  The  modifications  consisted  of 
monitoring  a  network  of  sensors;  retrieving  the  positions,  status,  and  data  associated 
with  each  sensor;  and  then  displaying  that  information  in  real  time  on  a  3-D  map. 
We  are  calling  the  result  a  3-D  Sensor  Common  Operating  Picture  (3-D  Sensor 
COP).  To  test  the  3-D  Sensor  COP,  we  took  advantage  of  a  sensor  network  that  had 
been  deployed  for  the  Enterprise  Challenge  2016  (EC  16)  at  Fort  Huachuca  in  Sierra 
Vista,  Arizona.  The  sensor  network  consisted  of  approximately  30  US  and 
Canadian  sensors  of  various  types  (proximity,  acoustic,  visible/IR  cameras), 
distributed  on  the  ground  across  a  test  range,  and  an  airborne  sensor  that  transmitted 
the  positions  of  moving  objects  on  the  test  range  down  to  a  ground  station.  Using 
the  data  from  the  sensor  network,  we  were  able  to  demonstrate  the  advantage,  from 
the  point  of  view  of  the  user,  of  seeing  real-time  data  integrated  with  the  3-D  terrain 
data. 

Fusion3D  contains  novel  features  not  provided  in  comparable  software  applications 
for  visualizing  3-D  terrain  data.  The  first  novel  feature  is  stereoscopic  viewing  of 
3-D  data.  Other  tools  display  3-D  data  on  a  2-D  screen,  thus  limiting  the  benefit  of 
the  3-D  data.  Fusion3D  uses  a  3-D  display  and  3-D  glasses  to  view  the  data  in 
stereo.  This  allows  users  to  experience  3-D  data  in  a  more  realistic  way  and 
provides  them  with  an  improved  understanding  of  the  terrain.  The  next  novel 
feature  is  that  it  displays  the  orthophoto  data  in  full  resolution.  The  orthophoto  data 
are  applied  as  a  texture  over  the  3-D  data  that  are  provided  by  LiDAR.  The  typical 
pixel  size  of  LiDAR-derived  Digital  Elevation  Model  (DEM)  data  is  1  m;  however, 
the  pixel  sizes  of  the  corresponding  orthophoto  data  can  be  much  smaller  (for 
example,  BuckEye  orthophoto  data  have  10-  to  20-cm  pixels).  Many  other  software 
applications  downsample  the  orthophoto  data  to  some  degree,  thereby  reducing  the 
spatial  content  of  the  orthophoto  data.  The  result  of  this  is  that  small  objects 
disappear  and  the  displayed  3-D  terrain  looks  less  realistic.  Fusion3D  retains  the 
full-resolution  imagery  by  applying  2-D  tiles  of  orthophoto  pixels  at  the  elevations 
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and  angles  indicated  by  the  LiDAR  DEM  data,  thus  providing  a  more  realistic  view 
of  the  scene  and  more  useful  information  to  the  user.  Fusion3D  also  has  a  patented 
method  for  applying  the  texture  where  locally  smooth  regions  are  joined  with 
adjacent  regions  in  a  technique  called  draping  (this  is  what  other  applications  use) 
and  regions  identified  as  rough  are  left  disconnected  from  their  neighbors 
(Dammann  2012).  This  novel  texturing  method  gives  natural  features,  such  as 
bushes  and  trees,  a  more  realistic  appearance. 

Fusion3D  is  capable  of  visualizing  data  from  a  variety  of  sensors  and  3-D 
reconstruction  approaches.  For  example,  Fusion3D  has  been  optimized  for  reading 
and  displaying  DEMs  derived  from  FiDAR  and  othophotos  co-collected  by  the 
BuckEye  program;  however,  it  is  also  capable  of  using  other  FiDAR  data,  such  as 
that  from  the  High  Altitude  FiDAR  Operations  Experiment  (HAFOE)  and  Airborne 
Faser  Detection  and  Radar  Imaging  Research  Testbed  (AFIRT)  programs. 
Fusion3D  can  visualize  either  the  raw  FiDAR  point  clouds  (in  .las  format)  or  the 
DEMs  that  have  been  computed  from  the  point  clouds.  Additionally,  Fusion3D  can 
also  display  3-D  data  created  using  photogrammetry  software.  Photogrammetry  is 
the  3-D  reconstruction  of  an  area  from  images,  using  a  structure-from-motion 
algorithm,  and  is  used  to  reconstruct  the  geometry  of  a  static  scene,  given  a  moving 
camera  perspective  (Spicer  2016).  Again,  the  display  of  the  3-D  data  can  be  done 
using  photogrammetry-derived  point  clouds  or  by  generating  DEMs  from  those 
point  clouds. 

Fusion3D  contains  a  variety  of  useful  features  that  can  assist  users  in  doing  their 
jobs.  For  example,  missions  can  be  planned  by  overlaying  potential  routes  on  the 
terrain  along  with  the  labels  of  significant  landmarks.  A  fly-through  can  then  be 
performed,  where  the  point  of  view  is  moved  to  the  start  of  the  planned  route  and 
then  the  route  is  followed.  This  gives  those  viewing  the  fly-through  some  idea  of 
the  appearance  of  the  terrain,  as  well  as  the  relative  positions  of  the  landmarks,  prior 
to  departing  on  the  mission.  Measurements  can  be  made  of  distances  or  elevation 
differences  between  points,  as  well  as  distances  along  a  route.  Additionally,  a  line- 
of-sight  tool  can  be  used  to  shade  areas  that  are  visible  from  a  selected  point  on  the 
map.  These  features  allow  different  routes  to  be  compared  according  to  factors  such 
as  total  distance,  change  in  elevation,  proximity  to  an  enemy  position,  or  the 
vulnerability  of  the  route. 

2.  The  3-D  Sensor  COP 


The  objective  of  the  3-D  Sensor  COP  is  to  provide  an  interactive  operator  interface 
for  visualizing  real-time  sensor  data  overlaid  on  a  3-D  terrain  map  for  enhanced 
situational  awareness.  Fusion3D  has,  therefore,  been  modified  to  incorporate 
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additional  features  so  that  it  can  be  used  as  a  real-time  COP.  Those  modifications 
include  the  ability  to  1)  read  sensor  data  that  have  been  collected  and  stored  on  a 
private  network,  2)  show  sensors  at  the  correct  positions  on  the  map,  and  3)  show 
the  status  of  the  sensors  and  the  sensor  data  that  have  been  collected. 

At  EC16,  FusionS D's  ability  to  perform  as  a  real-time  COP  was  demonstrated.  The 
3-D  terrain  map  that  was  used  for  the  exercise  was  a  BuckEye  map  collected  in 
September  2015.  Fusion3D  was  able  to  read  sensor  data  stored  as  extensible 
markup  language  (XML)  files  on  a  shared  folder  on  the  roll  on,  roll  off  (Ro-Ro) 
platform  via  a  private  local  area  network  and  then  show  the  positions  of  sensors 
that  had  been  deployed  in  the  field,  as  shown  in  Fig.  1. 


Fig.  1  Overhead  view  from  the  3-D  Sensor  COP  showing  sensor  locations 

A  user  was  then  able  to  interact  with  the  data  by  clicking  on  a  sensor  icon,  and 
information  about  that  sensor  was  displayed.  Any  data  that  are  included  in  the  XML 
file  for  each  sensor  can  potentially  be  displayed,  if  desired.  When  sensors  are 
triggered,  the  corresponding  sensor  data  are  shown  on  the  3-D  map.  The  sensor 
types  currently  supported  are  proximity  detectors,  acoustic  sensors  showing  the  line 
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of  bearing  of  a  detection,  and  imaging  sensors.  When  a  sensor  is  triggered,  its  icon 
changes  from  a  small  green  circle  to  a  large  red  circle,  as  shown  in  Fig.  2.  As  time 
passes,  that  red  circle  gradually  decreases  in  size  until  a  time  threshold  is  passed 
and  the  icon  becomes  green  again.  By  looking  at  the  sizes  of  the  red  circles,  a  user 
is  able  to  see  the  sequence  in  which  the  sensors  were  triggered.  This  gives  the  user 
an  idea  of  the  direction  of  travel  for  the  object  being  tracked. 


Fig.  2  Overhead  view  from  the  3-D  Sensor  COP  showing  sensor  locations  and 
observations 

Proximity  detections  are  shown  as  solid  red  circles  that  gradually  decrease  in  size. 
If  a  new  detection  does  not  occur,  they  will  eventually  turn  green  again,  indicating 
that  they  have  been  reset  back  to  a  listening  state.  If  the  sensor  data  include  a  line 
of  bearing,  that  bearing  is  a  shown  on  the  map  as  a  line  pointing  from  the  sensor 
position  toward  the  location  of  the  object  that  caused  the  detection,  as  shown  in 
Fig.  3.  These  directional  sensors  are  acoustic,  so  lines  point  in  the  direction  of  loud 
noises,  such  as  passing  vehicles  or  explosions. 
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Fig.  3  Observation  from  an  acoustic  sensor  showing  the  lines  of  bearing  of  the  detections 

For  an  image  sensor,  when  a  detection  occurs,  a  user  can  click  on  the  sensor  icon 
and  the  image  captured  by  that  sensor  is  displayed  on  the  map,  as  shown  in  Fig.  4. 
While  the  other  sensors  provide  evidence  that  an  object  is  near,  the  imaging  sensors 
give  the  user  the  opportunity  to  identify  the  object  or  the  activity  being  performed. 
For  example,  an  imaging  sensor  could  help  identify  an  enemy  placing  an 
improvised  explosive  device. 
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Fig.  4  Observation  from  an  imaging  sensor 

Additionally,  Fusion3D  was  modified  to  read  STANAG-4607  (20 10) -compliant 
tracks  transmitted  down  from  the  IR  motion  detection  (IRMD)  aircraft  and  place 
icons  indicating  moving  objects  on  the  3-D  map.  Fig.  5  shows  blue  icons  for 
detected  moving  objects.  The  moving  object  icons  will  gradually  decrease  in  size 
and  eventually  disappear  after  a  time  threshold  is  exceeded.  The  icons  along  the 
road  seem  reasonable,  while  those  off  the  road  may  be  false  detections.  With  some 
improvements  to  the  IRMD  moving  target  detection  algorithm,  a  user  would  be 
able  to  see  icons,  indicating  detected  moving  objects,  appear  in  areas  where  ground 
sensors  were  not  placed. 
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Fig.  5  Moving  objects  identified  by  the  IRMD  aircraft 


3.  Virtual  Sensor  Placement 


A  primary  focus  of  Fusion3D  is  its  ability  to  visualize  3-D  terrain,  and  one  area 
where  that  ability  is  useful  in  the  3-D  Sensor  COP  is  in  the  simulation  of  sensor 
placements.  Using  a  3-D  map,  a  user  can  zoom  in  on  a  potential  sensor  location  and 
get  an  idea  of  what  the  sensor  would  “see”.  This  would  allow  the  user  to  determine 
if  the  sensor  location  meets  the  desired  requirements  prior  to  undertaking  the 
mission  to  deploy  it.  For  example,  the  field  of  view  of  an  imaging  sensor  can  be 
checked  prior  to  placement  to  make  sure  that  the  desired  area  for  surveillance  is 
visible  and  free  of  obstructions.  This  functionality  makes  it  easier  to  plan  sensor 
networks  by  verifying  that  the  desired  coverage  is  achieved  with  as  few  sensors  as 
possible  and  reducing  the  number  of  times  sensors  have  to  be  moved  after 
placement.  Together,  this  reduces  the  risk  to  users  who  are  placing  the  sensor  by 
minimizing  the  amount  of  time  they  need  to  be  in  the  field. 
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Going  forward,  new  technologies  can  be  used  to  further  improve  virtual  sensor 
placement.  For  example,  using  either  LiDAR  or  photogrammetry  data,  collected  at 
or  near  ground-level,  a  3-D  model  of  the  potential  sensor  location  can  be  created. 
Additionally,  a  3-D  model  of  the  sensor  itself  can  be  created.  Using  these  3-D 
models,  along  with  emerging  virtual  and  augmented  reality  tools,  such  as  the 
Oculus  Rift  and  the  Microsoft  HoloLens  (Microsoft  2016),  a  user  can  virtually  enter 
the  potential  sensor  location  and  place  a  virtual  copy  of  the  sensor. 
Fig.  6  shows  an  example  from  the  National  Aeronautics  and  Space  Administration 
(NASA)  Jet  Propulsion  Laboratory  where  the  HoloLens  was  used  to  visualize  the 
surface  of  Mars  (NASA  2015). 


Fig.  6  NASA’s  Jet  Propulsion  Laboratory  demonstrating  the  use  of  the  HoloLens  to 
visualize  the  surface  of  Mars  (NASA  2015) 

Another  method  for  visualizing  terrain  and  virtually  placing  a  sensor  is  using  data 
collected  from  a  multi-camera  rig,  such  as  the  GoPro  Odyssey  (GoPro  2016),  which 
contains  16  synchronized  HER04  Black  cameras.  Rigs  such  as  this  can  capture 
360°  imagery  and  video  that  can  be  visualized  in  either  a  YouTube360  video  or  the 
HoloLens.  Figure  7  shows  images,  captured  by  the  onboard  HoloLens  camera,  of  a 
HoloLens  app,  published  by  Microsoft,  called  HoloTour  (Microsoft  2016),  during 
a  virtual  tour  of  Rome.  Similarly,  this  technique  could  be  used  to  acquire  and 
visualize  high-resolution  360°  imagery  of  sites  of  strategic  importance  to  the  Army. 
Three-dimensional  models  of  sensors  could  then  be  virtually  placed  and  evaluated 
in  an  accurate  representation  of  the  terrain. 
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Fig.  7  Two  images  from  a  virtual  tour  of  Rome,  captured  by  the  onboard  HoloLens  camera, 
while  running  the  Microsoft  HoloTour  app 
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4.  Conclusions 


The  modified  version  of  Fusion3D,  developed  for  EC  16  and  called  the  3-D  Sensor 
COP,  allows  a  user  to  visualize  sensor  data  collected  by  the  US  military  and  its 
partners,  while  at  the  same  time  visualizing  the  terrain  in  which  the  sensors  have 
been  placed.  For  example,  while  users  are  monitoring  a  large  area,  they  are  able  to 
see  sensors  being  triggered  in  a  local  region,  using  detections  from  either  ground- 
based  or  aerial  sensors.  They  can  then  follow  those  detections  and  click  on  the 
sensor  icons  to  get  more  information.  The  3-D  Sensor  COP  provides  a  way  for  users 
to  keep  track  of  a  large  number  of  sensors  that  have  been  placed  over  a  large  area. 
Additionally,  it  is  a  useful  tool  for  planning  where  to  place  sensors,  because  it 
makes  it  possible  to  see  the  terrain  from  multiple  locations  and  orientations  prior  to 
selecting  the  optimal  location  and  orientation  for  a  given  sensor. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


2-D 

2-dimensional 

3-D 

3-dimensional 

ALIRT 

Airborne  LaDAR  Imaging  Research  Testbed 

COP 

common  operating  picture 

DEM 

Digital  Elevation  Model 

EC16 

Enterprise  Challenge  2016 

IR 

infrared 

HALOE 

High  Altitude  LiDAR  Operations  Experiment 

IRMD 

infrared  motion  detection 

LaDAR 

Laser  Detection  and  Ranging 

LiDAR 

Light  Detection  and  Ranging 

Ro-Ro 

roll-on  roll-off 

XML 

extensible  markup  language 
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